(19) 



J) 




(12) 



(43) Date of publication: 

25.04.2001 Bulletin 2001/17 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets EP 1 094 337 A2 

EUROPEAN PATENT APPLICATION 

(51) lnt.CI. 7 : G01S 17/93 



(21) Application number: 00122303.1 

(22) Date of filing: 20.10.2000 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 
MCNLPTSE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 21.10.1999 J P 29963799 

(71) Applicant: 

MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD. 
Kadoma-shi, Osaka 571-8501 (JP) 



(72) Inventors: 

♦ Yasui, Nobuhiko 
Moriguchi-shl, Osaka-fu (JP) 

• lisaka, Atsushi 
Katano-shi, Osaka-fu (JP) 

(74) Representative: 

Lang, Johannes, Dipl.-lng. et al 

Bardehle Pagenberg Dost Altenburg Geissler 

Isenbruck, 

Postfach 86 06 20 

81633 Munchen (DE) 



(54) Parking assistance system 

(57) A parking assistance system is mounted in a 
vehicle, and comprises an image pick-up (10) with a sin- 
gle camera, an image processing device (20), a display 
(30), a steering angle sensor (40), a wheel speed sen- 
sor (50), and a pulse counter (60). An ADC (21) sub- 
jects two analog images picked up by the image pick-up 
(10) at different location to A/D conversion, and sends 
those to frame memories (22) and (23). A CPU (24) 



uses the digital image data and the transition data of the 
vehicle so as to calculate object data and a distance 
from the vehicle to the 3D object. An image thus 
obtained is then converted into a view from the above. 
The view from the above is stored in frame memory 
(27), and outputted to the display (30) under the control 
of a controller (28) for display. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to systems for assisting a driver of a vehicle to park by an image around the 
vehicle and, more specifically, to a system for calculating a distance from the vehicle to a specific 3D object in the image 
picked up from the vehicle and advising the driver of what are located around the vehicle. 

Description of the Background Art ' 

[0002] Conventional technologies so far disclosed relevant to vehicles, exemplarily automobiles, include detecting 
any obstacle around a vehicle, for example. If any obstacle is detected, and if collision therewith is determined as being 
highly possible, a driver of the vehicle is warned by alarm, advised of a distance to the obstacle, or according to the dis- 
tance, the vehicle is automatically braked to stop. As an example of such detection means, there is a distance detection 
system utilizing an image recognition technology, exemplarily disclosed in Japanese Patent Laid-Open No. 5-114099 
(93-1 1 4099). Utilized in this system is an Image picked up by a camera mounted in a vehicle. 

[0003] The distance detection system is now-described with reference to FIGS. 1 1 arid 12. Specifically, FIG. "1 1 is 
a diagram showing the relationship between a camera and a 3D object, while FIG. 12 a block diagram showing the dis- 
tance detection system. 

[0004] In the distance detection system, a stereo optical system is equipped in a vehicle, exemplarily in an automo- 
bile, for picking up an image of the 3D object located outside of the vehicle within a given range. The stereo optical sys- 
tem is exemplarily implemented by several cameras utilizing solid image-pickup device such as charge-coupled device 
(CCD), and each fixedly affixed to a ceiling surface of the vehicle toward the front. Here, the number of cameras is pre- 
sumably two, and the one located on the right, viewed from the front, is referred to as right camera, and the other on 
the left is as left camera. 

[0005] In FIG. 1 1, the right and left cameras are so placed as to be away from each other by s, and each photo- 
graph a point P at a perpendicular distance D therefrom. Assuming a focal distance for both cameras being /, an image 
of the point P is picked up on each camera's projection surface, which is away from the camera by the focal distance f. 
In this case, the distance between these two images on the projection surfaces is s + t, where f denotes a displace- 
ment. Thus, the distance D is so calculated as follows: 

D = s * f A 

That is, the displacement f leads to the distance D. 

[0006] Described next is the process of calculating the displacement t. In FIG. 12, a stereo optical system 1 00 for- 
wards two data of analog images picked up therein to an image processing part 201. The two analog image data are 
then subjected to A/D conversion for output to a hamming distance calculation part 202. The two digital image data are 
then calculated, respectively, for a hamming distance H on the pixel basis. The hamming distances H are used to deter- 
mine the displacement of two images. A minimum/maximum value detection part 203 then searches the hamming dis- 
tances H calculated by the humming distance calculation part 202 for a minimum value Hmin and a maximum value 
Hmax. It is herein assumed that the minimum value Hmin is the displacement t. 

[0007] As such, the distance detection system processes two images picked up by a stereo camera mounted in a 
vehicle, and finds a displacement t. Thereby, the system can assist a driver with a 3D position of an obstacle, for exam- 
ple, and a warning alarm if collision or bump thereagainst is possible. 

[0008] Such distance detection system, however, requires considerable accuracy for fixedly affixing several cam- 
eras of the stereo optical system to a vehicle especially in angle, direction, interval, and the like. There also needs to 
previously input, to a stereo image processing device, data indicating the satisfactory positions of those cameras. As a 
result, structuring such stereo optical system is very difficult. Further, as described in the foregoing, in order to calculate 
the hamming distances H, object detection needs to be done for one image with reference to the other in their entirety. 
Also, there needs to repeat such search on the pixel basis, the amount of calculation is thus quite large, and hardware 
structure is complicated. Still further; the distance to an obstacle advised by this distance detection system is often in 
numerical figures or by images being quite different from the actual obstacle. Therefore, the driver could not quite per- 
ceive the distance. 
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SUMMARY OF THE INVENTION 

[0009] Therefore, an object of the present invention is to provide a parking assistance system for assuredly advising 
a driver of a distance between his/her vehicle and an obstacle in a perceivable manner, while easing attachment of an 
image pick-up to the vehicle and reducing calculation in image processing. 
[0010] The present invention has the following features to attain the object above. 

[0011] An aspect of the present invention is directed to a parking assistance system for assisting a driver of a vehi- 
cle to park by picking up an image around the vehicle, and advising the driver of what 3D object is located therearound, 
the system comprising: 

an image pick-up for picking up, with a single camera during the vehicle's transition, first and second images around 
the vehicle picked up at different location and time of a first and a second; 

an object specification part for detecting and specifying the 3D object found in each of the first and second images; 
a vehicle position calculation part for calculating transition data of the vehicle from the first location to the second 
location; 

an object distance calculation part for calculating a distance from the vehicle to the 3D object by using the position 
of the 3D object in each of the first and second images, and the transition data of the vehicle; and 
an image creation part for creating a third image for display to the driver based on the images picked up by the 
- image- pick-up and data calculated by the object distance calculation part.- - - - - 

[0012] As described above, in the aspect of the present invention, as is provided with a single image pick-up, the 
parking assistance system picks up two images at different location and time, and calculates the transition data of the 
vehicle between the two locations. Therefore, with the principle of triangulation applied to the 3D objects in two images, 
the parking assistance system can correctly calculate the distance to the 3D object. Accordingly, one image pick-up will 
do, and the parking assistance system achieves easy attachment thereof as having no more need to fixedly affix several 
image pick-ups with considerable accuracy especially in angle, direction, interval, and the like. Further, the data calcu- 
lated by the object distance calculation part correctly indicates the distance from the vehicle, and thus the third image 
will not be askew and tell correct distance. Still further, by displaying the image onto which the distance data is added, 
the driver becomes easier to grasp the actual distance from the vehicle to the 3D object. 

[0013] The object distance calculation part preferably detects edges of the 3D objects each found in the first and 
second images. Also, the object distance calculation part uses the coordinate data of any edge identical in the first and 
second images and the transition data of the vehicle so that the distance from the vehicle to the edge of the 3D object 
is calculated. Therefore, even if the vehicle compiicatedly moves, the distance to the 3D object can be easily calculated 
only with the coordinates data of any edge identical in the two images and transit data of the vehicle. Further, in order 
to detect any identical edge, by using the transition data of the vehicle, the object distance calculation part may estimate 
the position of the edge identical in the first and second images, and then positionally subjects the edge to image anal- 
ysis. In this case, since the transition data of the vehicle has been previously calculated, the positional relationship 
between the two images is clarified. This makes possible to estimate which edge in one image is located at where in 
the other image, and image analysis is done only for the estimated part of the image, reducing the amount of calculation 
taken for the edge detection. Moreover, the hardware structure can be simplified. 

[0014] The image creation part preferably converts the image picked up by the image pick-up into a view from the 
above, and then creates a third image by adding the data calculated by the object distance calculation part onto the 
view. With the help of such created third image on the display, the driver can grasp the actual distance from the vehicle 
to the 3D object Alternatively, the image creation part may perceivably emphasizes, out of the data calculated by the 
object distance calculation part, a part in the vicinity of the vehicle. Thereby, out of the data calculated by the object dis- 
tance calculation part, a part in the vicinity of the vehicle may be emphasized by color, line width, or the like, to let them 
stand out to warn the driver against any approaching 3D object The image creation part also may generate the third 
image by converting the data calculated by the object distance calculation part into a 3D object model, and adding 3D 
object model onto the view from the above. As such, by adding the close-to-actual 3D object model onto the view from 
the above, the driver is warned against the 3D object, and also becomes easier to perceive the distance between the 
3D object and the vehicle. Or, the image creation part may additionally add a model of the vehicle onto the third image. 
As is equipped in the vehicle, the image pick-up cannot pick up an image of the vehicle in its entirety. In this sense, the 
image is not complete, and thus the driver may have hard time to grasp the actual distance from the vehicle to the 3D 
object. To get around such problem, the model of the vehicle is fixedly displayed on the third image. 
[0015] The object specification part, exemplarily, generates a fourth image by projecting the first image onto the 
second location for transformation, and then takes a difference between the fourth and second images to detect and 
specify the 3D object Alternatively, the object specification part may generate the fourth image by projecting the second 
image onto the first location for transformation, and then takes a difference between the fourth and first images to detect 
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and specify the 3D object. Either structure will do to detect and specify the 3D object in the image in an easy manner. 
This is because, due to extraction of such differences between one image actually picked up and the other obtained by 
projecting the former image thereonto, only any askew part is left behind. 

[001 6] The transition data of the vehicle typically includes the linear transition distance and the vehicle angle of the 
vehicle. In such case, based on the data based obtained from a steering angle sensor and wheel speed sensor via a 
pulse counter equipped in the vehicle, the vehicle position calculation part calculates the transit distance and the vehicle 
angle. 

[0017] The image pick-up is preferably equipped at the rear part of a vehicle having many blind spots especially at 
parking, thereby rendering the parking assistance system more useful. This is especially applicable to a large-sized 
vehicle. 

[001 8] In a preferable example, the above-described third image is displayed on the display for the driver. Since the 
third image looks quite close to the actual image, allowing the driver to easily perceive the actual distance. 
[0019] Further, in another preferable example, when the distance from the vehicle to the 3D object calculated by 
the object distance calculation part is shorter than a predetermined distance, a sound and/or light is generated. 
Thereby, the driver can be warned against the approaching 3D object. 

[0020] The image pick-up may be conditionally started in various manners. Firstly, the system may start picking-up 
an image in response to the manual operation of the driver. If this is the case, the driver can selectively start the parking 
assistance system only when he/she needs its assistance. Secondly, the system starts when the vehicle decelerates to 
a predetermined speed. Thirdly, the system starts when the vehicle goes"rearward. According to the second arid third 
conditions, the vehicle can tell whether being in a parking mode or not. Thus, the parking assistance system can be 
started only when the vehicle is in the parking mode. 

[0021] These and other objects, features, aspects and advantages of the present invention will become more 
apparent from the following detailed description of the present invention when taken in conjunction with the accompa- 
nying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[6022] 



FIG. 1 is a diagram showing the entire structure of a parking assistance system according to an embodiment of the 
present invention; 

FIG. 2 is a circuit block diagram of the parking assistance system; 

FIG. 3 is a flowchart showing the comprehensive operation of the parking assistance system; 
FIG. 4 is a flowchart showing the specific operation for object specification by the parking assistance system; 
FIGS. 5a to 5d are diagrams for assistance of explaining image processing executed for object specification by the 
parking assistance system; 

FIG. 6 is a flowchart showing the operation for calculating a distance to a 3D object by the parking assistance sys- 
tem; 

FIGS. 7a and 7b are diagrams for assistance of explaining edge detection in the 3D object by the parking assist- 
ance system; 

FIG. 8 is a diagram for assistance of explaining a method of calculating a distance to the 3D object when a driver 
has the vehicle go straight; 

FIG. 9 is a diagram for assistance of explaining a method of calculating a distance to the 3D object when the driver 
wheeled the vehicle; 

FIGS. 1 0a to 1 0c are diagrams for assistance of explaining a method of image creation for advising the diver of the 
distance to the 3D object; 

FIG. 11 is a diagram showing the relationship between a camera and a 3D object by a conventional distance detec- 
tion system disclosed in Japanese Patent Laid-Open No. 5-1 14099; and 
FIG. 12 is a block diagram showing the conventional distance detection system. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0023] FIGS. 1 to 10c are diagrams showing a parking assistance system according to an embodiment of the 
present invention. Specifically, FIG. 1 shows the entire structure, FIG. 2 a circuit block diagram, FIG. 3 a flowchart for 
the comprehensive operation, and FIG. 4 a flowchart for the operation of specifying a 3D object. Herein, the 3D object 
exemplarily includes a pillar, wall, another vehicle, and stroller located around a vehicle, and is defined as not including 
the ground or a white line drawn thereon. FIGS. 5a to 5d are diagrams for assistance of explaining image processing 
for specifying the 3D object, FIG. 6 a flowchart for the operation of calculating a distance to the 3D object, FIGS. 7a and 
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7b diagrams for assistance of explaining edge detection in the 3D object, FIG. 8 a diagram for assistance of explaining 
a method of calculating a distance to the 3D object when a driver let the vehicle go straight, FIG. 9 a diagram for assist- 
ance of explaining a method of calculating a distance to the 3D object when the driver wheeled the vehicle, and FIGS. 
1 0a to 1 0c diagrams for assistance of explaining a method of image creation for advising the diver of the distance to the 

5 3D object. With reference to FIGS. 1 to 10c, described below is an embodiment of the present invention. 

[0024] In FIG. 1, the parking assistance system of this embodiment is mounted in a vehicle 1 such as an automo- 
bile, and includes an image pick-up 10, an image processing device 20, a display 30, a steering angle sensor 40, a 
wheel speed sensor 50, and a pulse counter 60. The parking assistance system is also provided with a start controller 
(not shown) for controlling its own start. The image pick-up 10 is exemplarily implemented by a single camera utilizing 

w a solid image pickup device such as charge-coupled device (CCD), and such camera is so mounted In the vehicle 1 as 
to face a rearward direction. The image picked up by the camera is inputted into the image processing device 20. The 
steering angle sensor 40 is equipped to detect data indicating a wheel angle of the vehicle 1, and the steering angle 
data detected thereby also goes to the processing device 20. The wheel speed sensor 50 is the one provided to detect 
a distance traveled by the vehicle 1, and obtained data which indicates the number of wheel rotations (hereinafter, 

15 wheel rotation data) is inputted, via the pulse counter 60, into the image processing device 20. 

[0025] The image processing device 20 uses such data to calculate data having specified what 3D object is behind 
the vehicle 1 (hereinafter, object data), and then a distance between the specified 3D object and the vehicle 1 . The 
image processing device 20. then converts the image provided by the camera into a view from the above in order to help 
a driver "easily perceive the distance to the 3D object. And therebnto, the" calculated object data and distance data is 

20 both added. The image obtained thereby is then displayed on the display 30. The display 30 may be equipped with a 
speaker or a light-emitting device, for example, to warn the driver when the vehicle 1 approaches the 3D object too 
close, that is, when the distance between the vehicle 1 and the 3D object becomes less than a predetermined thresh- 
old. 

[0026] Described next is the internal structure of the image processing device 20. In FIG. 2, the image processing 
25 device 20 includes an analog/digital converter (hereinafter, ADC) 21 , frame memories 22, 23, and 27, a CPU 24, RAM 
25, ROM 26, and a controller 28. The ADC 21 converts an analog image picked up by the image pick-up 10 into a digital 
image, and provides the digital image data to the frame memory 22 or 23. The frame memories store the digital image 
data, and also forward the data to the CPU 24. Here, the digital image data stored in the frame memories 22 and 23 are 
each different image data picked up at each different position. The ROM 26 previously stores programs for image con- 
30 version and for calculating a positional displacement of the vehicle 1 , and the like. The program is loaded on the RAM 
25, which is used as a working area for calculating the positional displacement, for example. The CPU 24 uses all of 
the digital image data, steering angle data, and the wheel rotation data so as to go through image processing following 
the program on the RAM 25. In this image processing, the object data and the distance from the vehicle 1 to the 3D 
object is both calculated. The digital image provided by the frame memory 22 or 23 is then converted into a view from 
35 the above, onto which the calculated two data is to be added. The converted image is stored in the frame memory 27, 
and then outputted to the display 30 under the control of the controller 28 for display. 

[0027] Described next is the comprehensive operation of the image processing device 20 with reference to a flow- 
chart Details about steps are left for later description. In FIG. 3, the CPU 24 determines whether the driver turned ON 
the parking assistance system with a starting switch SW, which is the start controller (step S110). If Yes, the CPU 24 

40 starts its operation, and the procedure goes to step SI 20. If No, the parking assistance system is not started. Then, the 
CPU 24 specifies any 3D object in an image showing an area behind the vehicle (step S120), and then calculates the 
distance from the vehicle to the specified 3D object (step S130). The CPU 24 converts the image showing the area 
behind the vehicle into a view from the above (step S140), and thereon, the CPU 24 adds the object data and the dis- 
tance data. The 3D object is then displayed with emphasis given in the image thus obtained to let the driver notice (step 

45 S150). Lastly, the CPU 24 determines whether the driver turned OFF the starting switch SW (step S160). If Yes, this is 
the end of the operation of the parking assistance system, but if not yet, the procedure repeats step S120 and onward. 
[0028] Note herein that, the start controller may detect a reverse signal in a transmission of the vehicle or deceler- 
ation of the vehicle more than a predetermined threshold value, and in response thereto, the parking assistance system 
may automatically start. Further, the parking assistance system may be determined as being OFF in step S160 when 

so the CPU 24 detects the vehicle's reaching the speed higher than a predetermined threshold value. In this manner, the 
parking assistance system is set to automatically end its operation. 

[0029] Next, the details about the steps are described stepwise. First, with reference to FIGS. 2, 4, and 5a to 5d, it 
is described how to specify a 3D object in step S1 20 in FIG. 3. 

[0030] As already described, the flowchart of FIG. 4 shows the subroutine of step S120, how to specify a 3D object. 
55 Referring to FIGS. 2 and 4, the image pick-up 10 sends an analog image A to the ADC 21 . The analog image A repre- 
sents a vehicle at a position a. The ADC 21 then subjects the analog image A to A/D conversion so as to store thus 
obtained digital image A in the frame memory 22 (step S121). The CPU 24 uses, for calculation, steering angle data 
received from the steering angle sensor 40 and wheel rotation data from the wheel speed sensor 50, which are both 
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equipped in the vehicle. Here, calculated is a linear transit distance AD from the position a to b and a vehicle angle R 
during the transition (step S122). The CPU 24 then determines whether the linear transit distance AD is a predeter- 
mined threshold value or larger (step S123). If No, the procedure repeats step S122, and otherwise goes to step S124. 
The image pick-up 1 0 then sends an analog image B representing the vehicle at a position b to the ADC 21 . The ADC 
5 21 subjects the analog image B to A/D conversion so as to store thus obtained digital image B in the frame memory 23 
(step S124). 

[0031] For step S125 and onward, FIGS. 5a to 5d are referred to. FIG. 5a indicates the image A and FIG. 5b the 
image B, each including a 3D object 2 and white lines 3a and 3b drawn on the ground. The CPU 24 uses the linear tran- 
sit distance AD and the vehicle angle R calculated in step SI 22 for projective transformation. That is, the image A rep- 

10 resenting the vehicle at the position a is projected onto the image B at the position b, and an Image C is generated (step 
S125). To be more specific, the CPU 24 uses the data relevant to the vehicle's transition to subject the image A repre- 
senting the vehicle at the position a to image conversion, and thus obtains such image C as shown in. FIG. 5c, which is 
a view from the position o. By way of comparison, the white lines 3a and 3b in FIG. 5c are so converted as to coincide 
with those in FIG. 5b, but the 3D object 2 in FIG. 5c looks askew. This is because, in the projective transformation using 

15 the image A, the 3D object 2 is regarded as being on the same plane as the white lines 3a and 3b. 

[0032] The CPU 24 then calculates a difference between the images B and C (step S126). FIG. 5d shows the 
resultant image after the calculation, and reference numerals 4a and 4b therein denote regions, which are the differ- 
ences between the 3D objects 2 in the images B and C. The white lines 3a and 3b are not remained by extracting such 
-differences. That means, "due to extraction of the "difference's between the images ~B "an dC, only the 3D object 2 are left 

20 behind. The CPU 24 then outputs the object data as specified above (step S127). 

[0033] With reference to FIG. 2 and FIGS. 6 to 8, described next is step S120, calculating the distance from the 
vehicle to the 3D object. Herein, as already mentioned, FIG. 6 is a subroutine flowchart for calculating such distance. 
[0034] In FIGS. 2 and 6, the CPU 24 subjects the 3D object specified in the images A and B to edge detection by 
utilizing data of image A in the frame memory 22, data of image B in the frame memory 23, and the object data (step 

25 s131). In FIGS. 7a and 7b, the specified 3D object is a vehicle 5 exemplarily subjected to edge detection. As to the 
image representing the vehicle 5, analyzed first is a luminance distribution pattern (may be colored distribution pattern) 
and then difference in luminance among pixels. Determination is then made whether the difference in luminance is 
equal to or larger than a predetermined threshold value. In this manner, edges 6a to 6f are detected as shown in FIG. 
7a. FIG. 7b shows such detected edges 6a to 6f for the sake of clarity. As such, the CPU 24 subjects the 3D object spec- 

30 ified by the images A and B to edge detection, and then the procedure goes to step S1 32. 

[0035] In FIG. 6, the CPU 24 refers to the edges detected in the images A and B in step S1 31 to detect any identical 
edge in pairs (step S132). This edge detection is carried out under the method conventionally used to calculate the 
hamming distance, for example, by comparing the images A and B in luminance or color pattern. Note that, the CPU 24 
can estimate which edge in the image A is located at where in the image B with the help of the linear transit distance 

35 AD and the vehicle angle R of the vehicle. Accordingly, the CPU 24 goes for image analysis only for the estimated part 
of the image, and the amount of calculation taken for the edge detection is thus reduced. 

[0036] Next, the CPU 24 calculates the distance in each of the images for the identical edges detected in step S132 
(step S1 33). The distance is from a center position of each of the Images to each of those edges, and indicated by coor- 
dinates, for example. 

40 [0037] As to step S134, FIGS. 8 and 9 are referred to. FIG. 8 shows the method of calculating the distance from the 
vehicle to the 3D object when the driver has the vehicle go straight ahead (or go backward), in other words, when the 
vehicle moves to the Z direction without being wheeled. The method is described next below with reference to FIG. 8. 
[0038] In FIG. 8, assuming that the method is the one for calculating a distance D from the vehicle equipped with 
the image pick-up 10 to a point P on an edge of a 3D object 7. Herein, reference numerals 7a and Pa indicate where 

45 the 3D object 7 and the point P are to be each positioned when the vehicle is located at the position a. Similarly, refer- 
ence numerals 7b and Pb indicate where the 3D object 7 and the point P are to be each positioned when the vehicle is 
located at the position b. In FIG. 8, the position of the image pick-up 10 is taken as a reference, and shows the position 
change of the 3D object 7 and the point P. Specifically, if the vehicle transits from the position a to b by the linear transit 
distance AD, the 3D object 7 and the point P move from 7a and Pa to 7b to Pb, respectively. In FIG. 8, the 3D object 7 

so should have been taken as a reference so that the vehicle equipped with the image pick-up 1 0 is described as moving 
in the 2 direction by the linear transit distance AD. For the sake of convenience, however, the image pick-up 1 0 is taken 
as the reference so that the 3D object 7 and the point P are the ones moving by the linear transit distance AD. 
[0039] Assuming a focal distance for the image pick-up 10 being f, images of the 3D object 7 and the point P are 
projected onto a projection surface, which is away from the image pick-up 10 by the focal distance f. That means, the 

55 images of the 3D object 7 and the point P positioned at 7a and Pa, respectively, are projected on the image A as a 3D 
object 7'a and a point P'a. Similarly, those positioned at 7b and Pb are projected on the image B as a 3D object 7'b and 
a point P*b. Although the image shown in FIG. 8 is the combination of the images A and B for convenience, it is not prac- 
tically necessary in the operation of the parking assistance system. 
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[0040] Described next is the method of calculating the distance D from the vehicle to the point P on the edge of the 
3D object 7. Assuming that each center of the images A and B picked up by the image pick-up 1 0 is O, the distance D 
is expressed as 

5 D = AD * u/v 

where a distance from the center O.to the point P'a is u, and a distance from the point P'a to P'b is v. Here, since the 
linear transit distance AD of the vehicle has already been calculated in step-S122, the CPU 24 only needs to calculate 
the distance from the point P'a to P'b in each image to obtain the distance between the image pick-up 1 0 to the point P. 
io Note herein that, since the- edge is the collection of points, calculating the distance to the point P for such edge leads 
to the distance distribution of the edges. 

[0041] FIG. 9 is a diagram for assistance of explaining a method of calculating a distance from the vehicle to the 3D 
object when the driver wheeled the vehicle. The method is described next below. 

[0042] In FIG. 9, the vehicle 1 is assumed to move from the position a to b by the linear transit distance AD at the 
15 vehicle angle of R (not shown). Reference numerals 1 a and 1 b therein indicate the vehicle 1 being at the position a and 
b, respectively. The vehicle 1 is equipped with the image pick-up 1 0 whose coordinate optical axes at the position a are 
Xa, Ya, and 2a, and those at the position b are Xb, Yb, and 2b. Herein, ail of X, Y, and 2 denote directions, specifically 
2 denotes direction of the optical axis, X horizontal, and Y vertical. The 3D object to calculate the distance here is the 
point P on the edge of the 3D object 7. Next below, described is the method of calculating a distance Db from the vehicle 
20 1 b to the point P. 

[0043] As already described, the CPU 24 has calculated the vehicle angle R and the linear transit distance AD of 
the vehicle in step S122. Therefore, an equation can be established if the vehicle angle R is expressed by orthonorma! 
matrix of 3 by 3 and the linear transit distance AD by offset vector. 
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35 AD=(r 14 r 24r34 ) 

[0044] Here, presumably, in the image A picked up by the image pick-up 10 at the position a, coordinates of the 
point P are (x'a, ya), and those In the image B are (x'b, y'b). Moreover, the following equations are established, where 
the focal distance of the image pick-up 1 0 is presumed to be f, a distance from the image pick-up 1 0a to the point P is 
4o Da, and that from the image pick-up 1 0b to the point P is Db: 

(rn^- 12 yif + r 13 )Da + r 14 =^Db 

45 

( r 21 ^ + r 22 + r23 ) 03 + " Db 

/ x • a v • a 

SO [ r 31 — + f32 ~T + 03 + F 34 = Db 

[0045] With these equations, the distance Db will be: 

(r 34 x * b-fr 14 )( r34 x - a+r^y - a+fr 33 ) 1 ^ ^ 
r n x • a+r 12 y • a+fr 13 -x *b(r 31 x • a+r^y • a+fr^)/ 34 
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[0046] As such, even if the driver wheels the vehicle, the CPU 24 first calculates the coordinates of the point P in 
each of the images, and then the distance from the vehicle to the point P utilizing the data which indicates the linear 
transit distance AD and the vehicle angle R. Since the edge is the collection of points, the CPU 24 calculates the dis- 
tance to the point P for such edge, leading to the distance distribution of the edges. 
5 [0047] As is known from the above, the CPU 24 calculates the distance from the vehicle to the edge utilizing the 
vehicle angle R and the linear transit distance AD (step SI 34). 

[0048] Next, the CPU 24 checks whether such edge detected in step S131 has been subjected to calculation of dis- 
tance from the vehicle (step S135). If Yes, the procedure goes to step S138, and if not yet, goes to step SI 36. 
[0049] The CPU 24 then checks if the not-yet-calculated edge is the one detected in step SI 32 as being the iden- 
10 tical (step S1 36). If Yes, the procedure repeats the flow starting from step SI 33. If the image has any edge not yet cal- 
culated edge and not find its pair, the procedure goes to step S137, and the edge is deleted from the image. The 
procedure then goes to step S138. 

[0050] Lastly, the CPU 24 outputs data indicating the distances from the vehicle to each of the edges and the shape 
of edges (step S1 38). This is the end of the flow. 

15 [0051] Described next is step S140 in FIG. 3, creating an image a which is a view from the above. In FIG. 2, by 
using the data of the image B stored in the frame memory 23, the CPU 24 subjects the image B to image conversion 
such as affine transformation so as to convert the image into the image a, a view from the above (step S140). As 
already described, although the 3D object in the image B converted askew in the image a, the CPU 24 already has 
specified and detected the 3D object. With the object data, the 3D object is thus deleted from the image a. 

20 [0052] Next, described is step S150 in FIG. 3, emphasizing the 3D object in the image a for display. FIGS. 10a to 
1 0c are diagrams for assistance of explaining such method of giving emphasis to the 3D object. 

[0053] FIG. 10a shows the image B picked up by the image pick-up 10 at the position b. The image B includes 3D 
objects 2a and 2b and a white line 3. In the above step S140, in the process of converting the image B into the image 
a viewed from the above, the 3D objects 2a and 2b become askew and not precise, and thus are deleted therefrom. 

25 Accordingly, the CPU 24 adds these 3D objects onto the image a while emphasizing the shape thereof in such manner 
as to tell the driver how far those 3D objects really are (step S1 50). In this example, the CPU 24 utilizes the data indi- 
cating the distance to the edge of the 3D object outputted in step S138 and the shape of edges so as to add these 3D 
objects onto the image a. Next below, the object data to be added onto the image a is described. 
[0054] Here, in step S130, the CPU 24 detects the edges of the 3D object, and then calculates the distance for 

30 every edge from the vehicle. In step S150, the CPU 24 adds data indicating the shape of edges and the distance from 
the vehicle to each of the edges onto the image a. FIG. 1 0b shows an image p obtained by adding edge data 80a and 
80b of the 3D objects 2a and 2b onto the image a. The edge data 80a and 80b correctly indicates the distance from the 
vehicle, and thus is displayed precisely in the image p. In the edge data 80a and 80b displayed in the image p, the CPU 
24 may emphasize edges, at the tip thereof, color, line width, or the like, in the vicinity of the vehicle so as to let them 

35 stand out to warn the driver. 

[0055] A vehicle model 90 indicates the position and dimension of the vehicle. Here, as is equipped in the vehicle, 
the image pick-up cannot pick up an image of the vehicle in its entirety. In this sense, the image p is not complete, and 
thus the driver may have hard time to grasp the actual distance from the vehicle to the edge data 80a and 80b. To get 
around such problem, the CPU 24 fixes the position of the vehicle model 90 on the display. This vehicle model 90 is 

40 created by previously setting data indicating where the image pick-up is equipped in the vehicle, and the dimension of 
the vehicle. The CPU 24 may use the data from the steering angle sensor 40 to display, on the image p, a guidance path 
for the vehicle and its estimated trail 91. Such created image p is outputted from the CPU 24 shown in FIG. 2, and is 
stored in the frame memory 27. 

[0056] FIG. 1 0c shows an image y, which makes the driver grasp the actual distance with ease compared with the 
45 image p. On the image y, 3D object models 81a and 81b are pasted instead of the edge data 80a and 80b. The 3D 
object models 81a and 81b are selected among from a collection of 3D object models including any possible vehicle, 
wall, pillar, stroller, and the like, expected at parking. Prior thereto, the CPU 24 determines the type of 3D object based 
on shape of edges, for example, indicated by the edge data 80a and 80b. Note herein that, any part difficult to identify 
its type may be displayed as an object region, or left as being the edge data. As such, as the image P being added with 
so the close-to-actual 3D object models, the driver becomes easier to perceive the distance between the 3D object and 
the vehicle. The CPU 24 may use the data from the steering angle sensor 40 to display, on the Image P, a guidance path 
for the vehicle and its estimated trail 91 . Such created image y is also outputted from the CPU 24 shown in FIG. 2, and 
is stored in the frame memory 27. 

[0057] Here, the driver may determine which image of P or yis to be outputted from the CPU 24, or the CPU 24 may 
55 automatically determine by seeing how the edge data and the 3D object model coincide with each other. 

[0058] As is known from the above, the parking assistance system of the embodiment is fixedly provided with a sin- 
gle image pick-up, which picks up two images at different time and location. The parking assistance system thus carries 
out triangulation with respect to a 3D object. Accordingly, the parking assistance system has no more need to fixedly 
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affix several image pick-ups with considerable accuracy especially in angle, direction, interval, and the like, and thus 
one image pick-up will do. Further, as previously calculates the transit distance or heading direction of the vehicle, the 
parking assistance system thus can estimate the position where the 3D object is displayed in the image. In this manner, 
the amount of calculation taken to search for any identical 3D objects in two images can be reduced, and hardware 
structure is simplified: Still further, in order to display object data for the driver, the parking assistance system converts 
an image showing around the vehicle into a view from the above without causing the image to be askew. Also in thus 
obtained image, the 3D object is located at a precise distance, allowing the driver to perceive the actual distance from 
the vehicle to the 3D object. Needless to say, the parking assistance system can be effectively utilized by mounting the 
image pick-up at the rear part of a large-sized vehicle, which has many blind spots. 

[0059] While the invention has been described in detail, the foregoing description is in all aspects illustrative and 
not restrictive. It is understood that numerous other modifications and variations can be devised without departing from 
the scope of the invention. 

Claims 

1 . A parking assistance system for assisting a driver of a vehicle (1 ) to park by picking up an image around the vehicle 
(1), and advising the driver of what 3D object (2) is located therearound, the system comprising: 

an image pick-up (10) for picking up, with a single camera during said vehicle's" transition, first and second 

images around the vehicle picked up at different location and time of a first and a second; 

an object specification part for detecting and specifying said 3D object (2) found in each of said first and second 

images; 

a vehicle position calculation part for calculating transition data of said vehicle (1 ) from said first location to said 
second location; 

an object distance calculation part for calculating a distance from said vehicle (1 ) to said 3D object (2) by using 
the position of said 3D object (2) in each of said first and second images, and the transition data of said vehicle 
(1); and 

an image creation part for creating a third image for display to the driver based on the images picked up by said 
image pick-up (10) and data calculated by said object distance calculation part. 

2. The parking assistance system according to claim 1, wherein said object distance calculation part 

detects an edge (6) of said 3D object (2) from each of said first and second images; and 

calculates the distance from said vehicle (1) to the edge (6) of said 3D object (2) by using coordinate data of 

any edge identical in said first and second images and the transition data of said vehicle. 

3. The parking assistance system according to claim 2, wherein said object distance calculation part 

estimates the position of said edge (6) identical in said first and second images by using the transition data of 
said vehicle (1), and 

detects said identical edge (6) by positionally subjecting the estimated edge (6) to image analysis. 

4. The parking assistance system according to claim 1 , wherein said image creation part 

converts the image picked up by said image pick-up (10) into a view from the above, and 

creates said third image by adding the data calculated by said object distance calculation part onto said view 

from the above. 

5. The parking assistance system according to claim 4, wherein said image creation part perceivably emphasizes, out 
of the data calculated by said object distance calculation part, a part in the vicinity of said vehicle (1). 

6. The parking assistance system according to claim 4, wherein said image creation part generates said third image 
by converting the data calculated by said object distance calculation part into a 3D object model (81 ), and by adding 
said 3D object model (81) onto said view from the above. 

7. The parking assistance system according to claim 4, wherein said image creating part additionally adds a model 
(90) of said vehicle (1) onto said third image. 
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8. The parking assistance system according to claim 1 , wherein said object specification part 

creates a fourth image by projecting said first image onto said second location for transformation, and 
detects and specifies said 3D object (2) by taking a difference between said fourth and second images. 

9. The parking assistance system according to claim 1 , wherein said object specification part 

creates a fourth image by projecting said second image onto said first location for transformation, and 
detects and specifies said 3D object (2) by taking a difference between said fourth and first images. 

1 0. The parking assistance system according to claim 1 , wherein the transition data of said vehicle (1 ) includes a linear 
transit distance and a vehicle angle of said vehicle (1 ), and 

said vehicle position calculation part calculates said transit distance and vehicle angle based on data obtained 
is from a steering angle sensor (40) and wheel speed sensor (50) via a pulse counter (60) equipped in said vehi- 

cle (1). 

1 1 . The parking assistance system according to claim 1 , wherein said image pick-up (1 0) is mounted in said vehicle (1 ) 
as to face a rearward direction. 

20 

12. The parking assistance system according to claim 1, further comprising a display (30) for displaying said third 
image to said driver. 

1 3. The parking assistance system according to claim 1 , further comprising a speaker for generating a sound when the 
25 distance from said vehicle (1 ) to said 3D object (2) calculated by said object distance calculation part is shorter than 

a predetermined distance. 

1 4. The parking assistance system according to claim 1 , further comprising a light emitter for generating light when the 
distance from said vehicle (1) to said 3D object (2) calculated by said object distance calculation part is shorter than 

30 a predetermined distance. 

1 5. The parking assistance system according to claim 1 , wherein said image pick-up (10) starts picking up an image in 
response to the manual operation of the driver of said vehicle (1 ). 

35 16. The parking assistance system according to claim 1, wherein said image pick-up (10) starts picking up an image 
when said vehicle (1 ) decelerates to be slower than a predetermined speed. 

17. The parking assistance system according to claim 1, wherein said image pick-up (10) starts picking up an image 
when said vehicle (1) goes rearward. 

40 



45 



50 



55 



10 



EP 1 094 337 A2 




11 



EP 1 094 337 A2 




EP 1 094 337 A2 



F I G. 3 






SPECIFY 3D OBJECT 










CALCULATE DISTANCE 
TO 3D OBJECT 








CREATE IMAGE a, WHICH 
IS A VIEW FROM THE ABOVE 



I 



• S120 



•S130 



• SI 40 



DISPLAY IMAGE a WITH 
3D OBJECT GIVEN 
EMPHASIS THEREIN 



•S150 




13 



EP 1 094 337 A2 



F I G. 4 



START 




TAKE IN DATA OF IMAGE 
A SUBJECTED TO A/D 
CONVERSION 



•S121 



CALCULATE LINEAR -TRANSIT 
DISTANCE AND VEHICLE 
ANGLE 



-S122 



IS \ S123 

LINEAR TRANSIT 
"DISTANCE EQUAL TO OR LARGER* 
THAN PREDETERMINED 
^THRESHOLD VALUE 
? 



YES 



TAKE IN DATA OF IMAGE 
B SUBJECTED TO A/D 
CONVERSION 



I 



-S124 



CREATE IMAGE C BY 
PROJECTING IMAGE A FOR 
CONVERSION 



I 



TAKE DIFFERENCE BETWEEN 
IMAGES B AND C 



OUTPUT OBJECT DATA 




END 



S125 



S126 



S127 



14 



EP 1 094 337 A2 




EP 1 094 337 A2 



F I G. 6 




SUBJECT 3D OBJECTS IN 
IMAGES A AND B TO EDGE 
DETECTION 



I 



DETECT . ANY. IDENTICAL . EDGE 
IN IMAGES A AND B 



2 



CALCULATE LINEAR TRANSIT 
DISTANCE AND VEHICLE 
ANGLE OF EDGE ON SCREEN 



I 



CALCULATE DISTANCE FROM 
VEHICLE TO EDGE 
UTILIZING LINEAR TRANSIT 
DISTANCE AND VEHICLE 
ANGLE 



•S131 



•S132 



■S133 



-S134 




16 



EP 1 094 337 A2 



FIG. 7 A 




FIG. 7 B 



6 c 




17 



EP 1 094 337 A2 



F I G. 8 




18 



EP 1 094 337 A2 



F I G. 9 



Y b 




19 



EP 1 094 337 A2 




20 



EP 1 094 337 A2 



FIG. 11 



P 




21 



EP 1 094 337 A2 




* 



o 
o 




22 



(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(12) 



(88) Date of publication A3: 

09.01.2002 Bulletin 2002/02 



(H) EP 1 094 337 A3 

EUROPEAN PATENT APPLICATION 

(51) intci7: G01S 5/16, B60Q 1/48 



(43) Date of publication A2: 

25.04.2001 Bulletin 2001/17 

(21) Application number: 00122303.1 

(22) Date of filing: 20.10.2000 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 
MCNLPTSE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 21.10.1999 JP 29963799 

(71) Applicant: MATSUSHITA ELECTRIC INDUSTRIAL 
CO., LTD. 

Kadoma-shi, Osaka 571-8501 (JP) 



(72) Inventors: 

• Yasui, Nobuhiko 
Moriguchi-shi, Osaka-fu (JP) 

• lisaka, Atsushl 
Katano-shi, Osaka-fu (JP) 

(74) Representative: Lang, Johannes, Dipl.-lng. et al 
Bardehle Pagenberg Dost Altenburg Geissler 
Isenbruck, Postfach 86 06 20 
81633 Munchen (DE) 



(54) Parking assistance system 

(57) A parking assistance system is mounted in a 
vehicle, and comprises an image pick-up (10) with a sin- 
gle camera, an image processing device (20), a display 
(30), a steering angle sensor (40), a wheel speed sensor 
(50), and a pulse counter (60). An ADC (21) subjects 
two analog images picked up by the image pick-up (10) 
at different location to A/D conversion, and sends those 



to frame memories (22) and (23). A CPU (24) uses the 
digital image data and the transition data of the vehicle 
so as to calculate object data and a distance from the 
vehicle to the 3D object. An image thus obtained is then 
converted into a view from the above. The view from the 
above is stored in frame memory (27), and outputted to 
the display (30) under the control of a controller (28) for 
display. 



CO 
< 

CO 
CO 



CL 
LU 



FIG. 1 



2 0 



3 0 




1 0 



IMAGE PROCESSING! 
1DEVICE 



4 0 




PULSE 
COUNTER 



6 0 



5 0 



Printed by Jouvo, 75001 PARIS (FR) 



BNSDOCID: <EP 1094337A3 I > 



EP 1 094 337 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



AppfleaUoft Numter 

EP 00 12 2303 



DOCUMENTS CONSfDETOD TO BE RELEVANT 



Category 



0,A 



Qtafion of document wtth Indication, where appropriate, 
of rotavard pq«"»~ij-~ 



US 5 307 136 A (SANEYOSHI KEIJI) 
26 April 1994 (1994-04-26) 

* column 3, line 30 - column 4, line 65; 
figure 5 * 

EP 0 740 163 A (MATSUSHITA ELECTRIC IND CO 
LTD) 30 October 1996 (1996-10-30) 

* abstract * 

DE 42 38 599 A (BLOCK HUBERT K) 
9 June 1994 (1994-Q&-09) 

* abstract; claim 1 * 



Th* pnamnt search report has bean drawn up for afJ dhkns 



Reiewnt 
to cfaim 



CLASSRCATION OF THE 



601S5/16 
B6001/48 



TECHNICAL FIELDS 



601S 
B60Q 



THE HAGUE 



21 November 2001 



Haffner, R 



CATEGORY OF CITED DOCUMENTS 

X : parttgjterty refavant P ttXan afana 
Y t psfflctitKty raJswrt tf cmbttsd vAtftfinother 
document of tho bst 



A. ^ - fc- ^ — -i - _ ■ 
O : non-wHttan d 



D: 
L: 



or 



undflfMng the Invcrnton 
i art. but published on. c 



cttsd In I to Mpflcsfiofi 
eftadteroewi— on» 



P: 



S - rnmibof of 1h* canto pttflf famfly, correspond) no. 



2 



EP 1 094 337 A3 



ANNJXTOTHE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 00 12 2303 



The European Patent Office Is In no way table tor the** pa/Hcutera which are merely ^ven for the purpose of Information. 

21-11-2001 



Patent document 
cited In search report 


Publication 
dafcp 


1 


Patent lamfly 
■nemberfs) 


1 Publication 
date 


US 5307136 


A 


26-04-1994 


jp 
jp 

DE 
GB 


3167752 B2 
5114099 A 
4235619 Al 
2261339 A ,B 


21-05-2001 
07-05-1993 
29-04-1993 
12-05-1993 


EP 0740163 


A 


30-10-1996 


EP 
OP 
JP 
US 


0740163 A2 
3081788 B2 
9016900 A 
5904725 A 


30-10-1996 
28-08-2000 

17- 01-1997 

18- 05-1999 


DE 4238599 


A 


09-06-1994 


DE 


4238599 Al 


09-06-1994 



al 



For moredetafts about this annax :sm OfffctaJ Journal 



of tha EurapMJi PatantOfflca, No. 12/82 



BNSOOCIO: <EP 1094337A3J_> 



